{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Image classification example\n",
    "\n",
    "This example script runs inference using a number of popular image classification models.  This script is included in the NVIDIA TensorFlow Docker containers under `/workspace/nvidia-examples`.  See [Preparing To Use NVIDIA Containers](https://docs.nvidia.com/deeplearning/dgx/preparing-containers/index.html) for more information.\n",
    "\n",
    "You can enable TF-TRT integration by passing the `--use_trt` flag to the script.  This causes the script to apply TensorRT inference optimization to speed up execution for portions of the model's graph where supported, and to fall back on native TensorFlow for layers and operations which are not supported.  See [Accelerating Inference In TensorFlow With TensorRT User Guide](https://docs.nvidia.com/deeplearning/dgx/integrate-tf-trt/index.html) for more information.                                                                                                                                                                                          \n",
    "\n",
    "When using TF-TRT, you can use the precision option (`--precision`) to control precision.  float32 is the default (`--precision fp32`) with float16 (`--precision fp16`) or int8 (`--precision int8`) allowing further performance improvements.                                                                                                 \n",
    "\n",
    "int8 mode requires a calibration step (which is done automatically), but you also must specificy the directory in which the calibration dataset is stored with `--calib_data_dir /imagenet_validation_data`.  You can use the same data for both calibration and validation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Models\n",
    "\n",
    "We have verified the following models.\n",
    "\n",
    "* MobileNet v1\n",
    "* MobileNet v2\n",
    "* NASNet - Large\n",
    "* NASNet - Mobile\n",
    "* ResNet50 v1\n",
    "* ResNet50 v2\n",
    "* VGG16\n",
    "* VGG19\n",
    "* Inception v3\n",
    "* Inception v4\n",
    "\n",
    "For the accuracy numbers of these models on the ImageNet validation dataset, see [Verified Models](https://docs.nvidia.com/deeplearning/dgx/tf-trt-user-guide/index.html#verified-models)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Usage\n",
    "\n",
    "The example Python script is `image_classification.py`.  You can evaluate inference with TF-TRT integration using the pre-trained ResNet V1 50 model by calling the script with the following arguments:\n",
    "\n",
    "```\n",
    "python image_classification.py --model resnet_v1_50 \\\n",
    "    --data_dir /path/to/imagenet/tfrecord/files \\\n",
    "    --use_trt \\\n",
    "    --precision fp16\n",
    "```\n",
    "\n",
    "Where:\n",
    "\n",
    "`--model`: Which model to use to run inference, in this case ResNet V1 50.\n",
    "\n",
    "`--data_dir`: Path to the ImageNet TFRecord validation files.\n",
    "\n",
    "`--use_trt`: Convert the graph to a TensorRT graph.\n",
    "\n",
    "`--precision`: Precision mode to use, in this case FP16.\n",
    "\n",
    "Run with `--help` to see all available options.\n",
    "\n",
    "Note: In this notebook, we run the script inside IPython using the `%run` built-in command, so that realtime output and tracebacks are displayed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%run image_classification --help"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also see [General Script Usage](https://docs.nvidia.com/deeplearning/dgx/tf-trt-user-guide/index.html#image-class-usage) for more information."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Output\n",
    "\n",
    "The script first loads the pre-trained model.  If given the flag `--use_trt`, the model is converted to a TensorRT graph, and the script displays (in addition to its inital configuration options):\n",
    "\n",
    "- the number of nodes before conversion (`num_nodes(native_tf)`)\n",
    "\n",
    "- the number of nodes after conversion (`num_nodes(trt_total)`)\n",
    "\n",
    "- the number of separate TensorRT nodes (`num_nodes(trt_only)`)\n",
    "\n",
    "- the size of the graph before conversion (`graph_size(MB)(native_tf)`)\n",
    "\n",
    "- the size of the graph after conversion (`graph_size(MB)(trt)`)\n",
    "\n",
    "- how long the conversion took (`time(s)(trt_conversion)`)\n",
    "\n",
    "For example:\n",
    "\n",
    "```\n",
    "num_nodes(native_tf): 741\n",
    "num_nodes(trt_total): 10\n",
    "num_nodes(trt_only): 1\n",
    "graph_size(MB)(native_tf): ***\n",
    "graph_size(MB)(tft): ***\n",
    "time(s)(trt_conversion): ***\n",
    "```\n",
    "\n",
    "Note: For a list of supported operations that can be converted to a TensorRT graph, see the [Supported\n",
    "Ops](https://docs.nvidia.com/deeplearning/dgx/integrate-tf-trt/index.html#support-ops) section of the [Accelerating Inference In TensorFlow With TensorRT User Guide](https://docs.nvidia.com/deeplearning/dgx/integrate-tf-trt/index.html).\n",
    "\n",
    "The script then begins running inference on the ImageNet validation set, displaying run times of each iteration after the interval defined by the `--display_every` option (default: `100`):\n",
    "\n",
    "```\n",
    "running inference...\n",
    "    step 100/6202, iter_time(ms)=**.****, images/sec=***\n",
    "    step 200/6202, iter_time(ms)=**.****, images/sec=***\n",
    "    step 300/6202, iter_time(ms)=**.****, images/sec=***\n",
    "    ...\n",
    "```\n",
    "\n",
    "On completion, the script prints overall accuracy and timing information over the inference session:\n",
    "\n",
    "```\n",
    "results of resnet_v1_50:\n",
    "    accuracy: 75.95\n",
    "    images/sec: ***\n",
    "    99th_percentile(ms): ***\n",
    "    total_time(s): ***\n",
    "    latency_mean(ms): ***\n",
    "```\n",
    "\n",
    "The accuracy metric measures the percentage of predictions from inference that match the labels on the ImageNet Validation set.  The remaining metrics capture various performance measurements:\n",
    "\n",
    "- number of images processed per second (`images/sec`)\n",
    "\n",
    "- total time of the inference session (`total_time(s)`)\n",
    "\n",
    "- the mean duration for each iteration (`latency_mean(ms)`)\n",
    "\n",
    "- the slowest duration for an iteration (`99th_percentile(ms)`)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using TF-TRT With ResNet V1 50\n",
    "\n",
    "Here we walk through how to use the example Python scripts in the with the ResNet V1 50 model.\n",
    "\n",
    "Using TF-TRT with precision modes lower than FP32, that is, FP16 and INT8, improves the performance of inference.  The FP16 precision mode uses Tensor Cores or half-precision hardware instructions, if possible, while the INT8 precision mode uses Tensor Cores or integer hardware instructions.  INT8 mode also requires running a calibration step, which the script does automatically.\n",
    "\n",
    "Below we use the example script to compare the accuracy and timing performance of all the precision modes when running inference using the ResNet V1 50 model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Native TensorFlow Using FP32\n",
    "\n",
    "This is our baseline session running inference using native TensorFlow without TensorRT integration/conversion.\n",
    "\n",
    "First, set `DATA_DIR` to where you stored the ImageNet TFRecord validation files:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "DATA_DIR = \"/path/to/imagenet/tfrecord/files\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can run the baseline session with native TensorFlow.\n",
    "\n",
    "Note: We use the `--cache` flag to allow the script to cache checkpoint and frozen graph files to use with future sessions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%run image_classification --model resnet_v1_50 \\\n",
    "    --data_dir $DATA_DIR \\\n",
    "    --cache"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Look for the accuracy and timing information under:\n",
    "\n",
    "```\n",
    "results of resnet_v1_50:\n",
    "    ...\n",
    "```\n",
    "\n",
    "You can compare the accuracy metrics for the ResNet 50 models with the metrics listed at: [Pre-trained model](https://github.com/tensorflow/models/tree/master/official/resnet#pre-trained-model)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TF-TRT Using FP32\n",
    "\n",
    "In this session, we use the same precision mode as in our native TensorFlow session (FP32), but this time we use the `--use_trt` flag to convert the graph to a TensorRT optimized graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%run image_classification --model resnet_v1_50 \\\n",
    "    --data_dir $DATA_DIR \\\n",
    "    --use_trt \\\n",
    "    --cache"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before the script starts running inference, it converts the TensorFlow graph to a TensorRT optimized graph with fewer nodes.  Look for the following metrics in the log:\n",
    "\n",
    "```\n",
    "num_nodes(native_tf): ***\n",
    "num_nodes(tftrt_total): ***\n",
    "num_nodes(trt_only): ***\n",
    "graph_size(MB)(native_tf): ***\n",
    "graph_size(MB)(tft): ***\n",
    "...\n",
    "time(s)(trt_conversion): ***\n",
    "```\n",
    "\n",
    "Note: For a list of supported operations that can be converted to a TensorRT graph, see [Supported Ops](https://docs.nvidia.com/deeplearning/dgx/integrate-tf-trt/index.html#support-ops).\n",
    "\n",
    "Again, note the accuracy and timing information under:\n",
    "\n",
    "```\n",
    "results of resnet_v1_50:\n",
    "    ...\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TF-TRT Using FP16\n",
    "\n",
    "In this session, we continue to use TF-TRT conversion, but we reduce the precision mode to FP16, allowing the use of Tensor Cores for performance improvements during inference, while preserving accuracy within the acceptable tolerance level (0.1%)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%run image_classification --model resnet_v1_50 \\\n",
    "    --data_dir $DATA_DIR \\\n",
    "    --use_trt \\\n",
    "    --precision fp16 \\\n",
    "    --cache"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, we see that the native TensorFlow graph gets converted to a TensorRT graph.  Look again for the following in the log to confirm:\n",
    "\n",
    "```\n",
    "num_nodes(native_tf): ***\n",
    "num_nodes(tftrt_total): ***\n",
    "num_nodes(trt_only): ***\n",
    "graph_size(MB)(native_tf): ***\n",
    "graph_size(MB)(tft): ***\n",
    "...\n",
    "time(s)(trt_conversion): ***\n",
    "```\n",
    "\n",
    "Compare the results with the previous sessions:\n",
    "\n",
    "```\n",
    "results of resnet_v1_50:\n",
    "    ...\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TF-TRT Using INT8\n",
    "\n",
    "For this session we continue to use TF-TRT conversion, and we reduce the precision further to INT8 for faster computation.  Because INT8 has significantly lower precision and dynamic range than FP32, the INT8 precision mode requires an additional calibration step before performing the type conversion.  In this calibration step, inference is first run with FP32 precision on a calibration dataset to generate many INT8 quantizations of the weights and activations in the trained TensorFlow graph, from which are chosen the INT8 quantizations that minimize information loss.  For more details on the calibration process, see the [8-bit Inference with TensorRT presentation](http://on-demand.gputechconf.com/gtc/2017/presentation/s7310-8-bit-inference-with-tensorrt.pdf).\n",
    "\n",
    "The calibration dataset should closely reflect the distribution of the problem dataset.  In this walkthrough, we use the same ImageNet validation set training data for the calibration data, with `--calib_data_dir $DATA_DIR`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%run image_classification --model resnet_v1_50 \\\n",
    "    --data_dir $DATA_DIR \\\n",
    "    --use_trt \\\n",
    "    --precision int8 \\\n",
    "    --calib_data_dir $DATA_DIR \\\n",
    "    --cache"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This time, we see the script performing the calibration step:\n",
    "\n",
    "```\n",
    "Calibrating INT8...\n",
    "...\n",
    "INFO:tensorflow:Evaluation [6/62]\n",
    "INFO:tensorflow:Evaluation [12/62]\n",
    "INFO:tensorflow:Evaluation [18/62]\n",
    "...\n",
    "```\n",
    "\n",
    "The process completes with the message:\n",
    "\n",
    "```\n",
    "INT8 graph created.\n",
    "```\n",
    "\n",
    "When the calibration step completes -- it may take some time -- we again see that the native TensorFlow graph gets converted to a TensorRT graph.  Look again for the following in the log to confirm:\n",
    "\n",
    "```\n",
    "num_nodes(native_tf): ***\n",
    "num_nodes(tftrt_total): ***\n",
    "num_nodes(trt_only): ***\n",
    "graph_size(MB)(native_tf): ***\n",
    "graph_size(MB)(tft): ***\n",
    "...\n",
    "time(s)(trt_conversion): ***\n",
    "```\n",
    "\n",
    "Also notice the following INT8-specific timing information:\n",
    "\n",
    "```\n",
    "time(s)(trt_calibration): ***\n",
    "...\n",
    "time(s)(trt_int8_conversion): ***\n",
    "```\n",
    "\n",
    "Compare the results with the previous sessions:\n",
    "\n",
    "```\n",
    "results of resnet_v1_50:\n",
    "    ...\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "Congratulations!  You have run inference with an image classification model using various modes of precision and taking advantge of TensorRT inference optimization where possible."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
